{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Part of Speech Tagging" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Part of speech tagging task aims to assign every word/token in plain text a category that identifies the syntactic functionality of the word occurrence.\n", "\n", "Polyglot recognizes 17 parts of speech, this set is called the `universal part of speech tag set`:\n", "\n", "- **ADJ**: adjective\n", "- **ADP**: adposition\n", "- **ADV**: adverb\n", "- **AUX**: auxiliary verb\n", "- **CONJ**: coordinating conjunction\n", "- **DET**: determiner\n", "- **INTJ**: interjection\n", "- **NOUN**: noun\n", "- **NUM**: numeral\n", "- **PART**: particle\n", "- **PRON**: pronoun\n", "- **PROPN**: proper noun\n", "- **PUNCT**: punctuation\n", "- **SCONJ**: subordinating conjunction\n", "- **SYM**: symbol\n", "- **VERB**: verb\n", "- **X**: other" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Languages Coverage" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The models were trained on a combination of:\n", "\n", "- Original CONLL datasets after the tags were converted using the [universal POS tables](http://universaldependencies.github.io/docs/tagset-conversion/index.html).\n", "\n", "- Universal Dependencies 1.0 corpora whenever they are available." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 1. German 2. Italian 3. Danish \n", " 4. Czech 5. Slovene 6. French \n", " 7. English 8. Swedish 9. Bulgarian \n", " 10. Spanish; Castilian 11. Indonesian 12. Portuguese \n", " 13. Finnish 14. Irish 15. Hungarian \n", " 16. Dutch \n" ] } ], "source": [ "from polyglot.downloader import downloader\n", "print(downloader.supported_languages_table(\"pos2\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Download Necessary Models" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[polyglot_data] Downloading package embeddings2.en to\n", "[polyglot_data] /home/rmyeid/polyglot_data...\n", "[polyglot_data] Package embeddings2.en is already up-to-date!\n", "[polyglot_data] Downloading package pos2.en to\n", "[polyglot_data] /home/rmyeid/polyglot_data...\n", "[polyglot_data] Package pos2.en is already up-to-date!\n" ] } ], "source": [ "%%bash\n", "polyglot download embeddings2.en pos2.en" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We tag each word in the text with one part of speech." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": false }, "outputs": [], "source": [ "from polyglot.text import Text" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ "blob = \"\"\"We will meet at eight o'clock on Thursday morning.\"\"\"\n", "text = Text(blob)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can query all the tagged words" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "[(u'We', u'PRON'),\n", " (u'will', u'AUX'),\n", " (u'meet', u'VERB'),\n", " (u'at', u'ADP'),\n", " (u'eight', u'NUM'),\n", " (u\"o'clock\", u'NOUN'),\n", " (u'on', u'ADP'),\n", " (u'Thursday', u'PROPN'),\n", " (u'morning', u'NOUN'),\n", " (u'.', u'PUNCT')]" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "text.pos_tags" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After calling the pos_tags property once, the words objects will carry the POS tags." ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false, "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "u'PRON'" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "text.words[0].pos_tag" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Command Line Interface" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "which DET \r\n", "India PROPN\r\n", "beat VERB \r\n", "Bermuda PROPN\r\n", "in ADP \r\n", "Port PROPN\r\n", "of ADP \r\n", "Spain PROPN\r\n", "in ADP \r\n", "2007 NUM \r\n", ", PUNCT\r\n", "which DET \r\n", "was AUX \r\n", "equalled VERB \r\n", "five NUM \r\n", "days NOUN \r\n", "ago ADV \r\n", "by ADP \r\n", "South PROPN\r\n", "Africa PROPN\r\n", "in ADP \r\n", "their PRON \r\n", "victory NOUN \r\n", "over ADP \r\n", "West PROPN\r\n", "Indies PROPN\r\n", "in ADP \r\n", "Sydney PROPN\r\n", ". PUNCT\r\n", "\r\n" ] } ], "source": [ "!polyglot --lang en tokenize --input testdata/cricket.txt | polyglot --lang en pos | tail -n 30" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "### Citation\n", "\n", "This work is a direct implementation of the research being described in the [Polyglot: Distributed Word Representations for Multilingual NLP](http://www.aclweb.org/anthology/W13-3520) paper.\n", "The author of this library strongly encourage you to cite the following paper if you are using this software." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "```\n", " @InProceedings{polyglot:2013:ACL-CoNLL,\n", " author = {Al-Rfou, Rami and Perozzi, Bryan and Skiena, Steven},\n", " title = {Polyglot: Distributed Word Representations for Multilingual NLP},\n", " booktitle = {Proceedings of the Seventeenth Conference on Computational Natural Language Learning},\n", " month = {August},\n", " year = {2013},\n", " address = {Sofia, Bulgaria},\n", " publisher = {Association for Computational Linguistics},\n", " pages = {183--192}, \n", " url = {http://www.aclweb.org/anthology/W13-3520}\n", " }\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## References\n", "\n", "- [Universal Part of Speech Tagging](http://universaldependencies.github.io/docs/u/pos/index.html)\n", "- [Universal Dependencies 1.0](https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-1464)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 0 }